Repository navigation
fix(writer): bit-packing and ALP write their patches as Rust does - #619
Merged
Merged
Conversation
Rust's gather_patches (fastlanes bitpack_compress/global.rs) records, for each 1024-value chunk, how many patches come before it, and the bit-packing and ALP schemes then run compress_patches (vortex-btrblocks schemes/patches.rs): indices and chunk offsets narrowed to the smallest unsigned type that holds their largest value, and values that are all equal (one value included) stored as a vortex.constant. Java wrote bit-packing's indices typed by the array length, no chunk offsets, and both encodings' values as a primitive, so a patched column held a different encoding tree from vortex-jni's. The patch layout of bit-packing, ALP and ALP-RD now goes through PatchBuffers; PatchBuffers.valuesNode is the constant-or-primitive rule, checking and encoding in one scan (ConstantEncodingEncoder.encodeIfConstant). Bit-packing hands its widened patch values straight through, written at their width in one copy, and ALP collects its exceptions into primitive arrays instead of boxed lists. Tests: NullableIntegerInteropIntegrationTest covers patched columns (wide and equal outliers), which #609 had to leave out, and NullableFloatInteropIntegrationTest F64 and F32 shapes with equal exceptions, with and without nulls; all fail without the change. Bit-packing patches round-trip for every integer ptype (U8 200 and U64 above 2^63 catch a sign-extension); a wide value at a null row is never patched, directly or through FoR over negatives; PatchBuffersTest covers chunk-boundary offsets and narrowing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfa1
force-pushed
the
fix/bitpacked-patch-chunk-offsets
branch
from
October 31, 2026 14:48
9360965 to
d15d023
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #609, which left patched columns out of its vortex-jni cross-checks: Rust stores bit-packing patches with chunk offsets and Java did not.
Two commits, one per bug:
gather_patchesrecords the number of patches before each 1024-value chunk, and the bit-packing scheme runscompress_patches(vortex-btrblocksschemes/patches.rs). That narrows the indices and chunk offsets to the smallest unsigned type that holds them, and stores equal values as avortex.constant. Java wrote the indices (typed by array length) and primitive values only. The patch layout now goes throughPatchBuffers, which ALP already used.compress_patches, so equal exceptions become avortex.constant. Java always wrote a primitive. ALP's four patch sites now use the samePatchBuffers.append.Tests
NullableIntegerInteropIntegrationTest:wide outliersandequal outliersshapes (patched bit-packing). Each asserts vortex-jni's encoding tree, then that vortex-jni reads Java's file back (default and compact; full, filtered, row-range). Both tree checks fail without commit 1.NullableFloatInteropIntegrationTest:f64 equal exceptionsandf32 equal exceptions. Both fail without commit 2../mvnw verifyis green locally.🤖 Generated with Claude Code